Skip to content

Retry package-set heap exhaustion with clean compiler output - #815

Open
thomashoneyman wants to merge 2 commits into
masterfrom
fix/package-set-heap-retry
Open

thomashoneyman wants to merge 2 commits into
masterfrom
fix/package-set-heap-retry

Conversation

@thomashoneyman

Copy link
Copy Markdown
Member

So, we’ve been getting health check alerts because the compiler runs out of memory while building a package set with 60 updates. The previous change (#813) stopped those failures from taking down the server, but the build still hit the compiler’s memory limit. Surprising, since I had run the build locally in the same circumstances (so I thought) seen on the server!

Unfortunately, my error was to reproduce building the whole set from a clean state, whereas what happens in an actual package set build is that we build the current package set, install the updates, then compile again using the existing output. In a new local reproduction that failed with heap exhaustion. Removing only the compiler output and rebuilding the same sources succeeded within the memory budget.

The compiler loads existing compiled module information before deciding which modules dependency changes force it to rebuild, so an incremental build can have different memory costs from a clean one. I haven’t pinned down the exact source of the extra memory, but clearing the output consistently resolved this build’s failure.

So: in this PR, we detect the explicit heap-exhaustion message and allow one retry with clean output for package-set builds. For typical builds you get an incremental, fast one. For big builds that would exhaust memory, you get a retry. It keeps the rollback backup intact, leaves ordinary package builds unchanged. We were also misclassifying heap exhaustion as a "compiler race” (the unexpected JSON output bug).

The app tests pass. I also ran the full sequence through the updated handler under a 3.5 GiB memory limit. It hit the same error, cleared the output, and successfully rebuilt all 60 updates on its one retry, without using swap. So now I’m feeling confident this will work as expected.

The alternative as always is to host this somewhere with more RAM, but for now, this is a reasonable stopgap IMO.

Comment thread app/src/App/CLI/Purs.purs
@thomashoneyman

Copy link
Copy Markdown
Member Author

@f-f I deployed this branch before merging and reran the same 60-package update on the server, leaving the existing compiler output in place.

It hit heap exhaustion, logged exactly one clean-output retry, and then succeeded. The run took about 20 minutes and published 81.8.0.

No new out of memory kills, server restarts, or job resets during the run. So this has now handled the actual production failure, not just the local reproduction.

Heres' the successful job

@f-f
f-f force-pushed the fix/package-set-heap-retry branch from a0b5e3c to 9c8503a Compare October 18, 2026 22:57
Comment thread app/src/App/CLI/Purs.purs
Comment thread app/src/App/Effect/PackageSets.purs Outdated
Comment thread app/test/App/CLI/Purs.purs Outdated
@thomashoneyman
thomashoneyman requested a review from f-f October 20, 2026 21:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants